I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

1. Refactoring TransformCompact:

   • The TransformCompact was split into two components to improve the modularity and efficiency of the compaction process:
     • BlockCompactBuilder: Responsible for constructing compaction tasks.
     • TransformCompactBlock: Executes the actual compaction in a parallelized manner.

2. Improvement in Compaction Logic:

   • The logic was adjusted to avoid writing excessively large data blocks during compaction.
   • The new structure ensures compaction is done as early as possible during the data writing phase.

3. Block Size Control:

   • The changes aim to fine-tune block sizes during the compaction, ensuring that the resulting blocks are neither too small nor too large, which can impact the efficiency of both storage and read operations.

4. Replace HashMap with BTreeMap in reclustering fetch_max_depth, for stable reclustering effects

5. Compact source data blocks before reclustering, for better performance and clustering

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

This change is